Inpainting is the process of reconstructing lost or deteriorated parts of images and videos. For instance, in the case of a valuable painting, this task would be carried out by a skilled image restoration artist. In the digital world, inpainting, also known as image interpolation or video interpolation, refers to the application of sophisticated algorithms to recover lost or corrupted parts of the image data.
Creating depth by annotation-and-propagation is the process of drawing a depth map for one image frame in an image sequence, i.e. annotating it, and then utilizing software tracking of the annotated objects to propagate the depth to the next/previous frames in the image sequence. This is a known method of semi-automated depth creation. Estimated depth information, independently of its source, is typically plagued with problems such as real object boundary misalignment.
FIG. 1a illustrates an image comprising an object, in which the depth misalignment with objects is noticeable as indicated by the bright areas protruding at the top and bottom beyond the dark area. FIG. 1a shows the depth component overlaid over luminance. This depth misalignment is due to the fact that a small portion of the foreground object is incorrectly considered background. Generally misalignment occurs between at least two signals/data, such as depth component and video data.
When compressing 2.5D videos utilizing commonly known lossy video compression schemes, artifacts surrounding objects are introduced. 2.5D (two-and-a-half dimensional) is an informal term used to describe a representation where a 2D image is accompanied by a depth map, which indicates the depth of each element in the 2D image.
Some artifacts caused by video compression are mosquito noise, Gibbs effect and overshoot/undershoot artifacts. FIG. 1b illustrates that halos are created around objects utilizing such lossy video compression schemes. This may occur both in 2D images or 2D videos.
The above-mentioned artifacts will generally result in noticeably decreased image quality, but they will also interfere with any subsequent video processing task, such as multi-view rendering from such a 2.5D representation.
Hence an improved method, processing unit, and computer program product allowing for removal of artifacts around objects caused by video compression, and/or correction of image data, which is misaligned with reference to depth maps when a small portion of the foreground object is incorrectly considered as background would be advantageous.